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Abstract 


In  this  project  we  adapted  the  newly  developed  Phase-Based  Video  Processing  (PBVP)  algorithm  to 
enhance  the  visibility  of  subtle  temporal  changes  (movement  or  temperature  fluctuations)  in 
multiband  (visual,  near-,  shortwave-  and  longwave-infrared)  imagery  while  simultaneously  reducing 
dynamic  noise.  We  successfully  applied  the  adapted  algorithm  to  enhance  the  visibility  of  small 
movements  in  the  Visual,  Near-Infrared  and  Thermal  (LWIR)  bands  of  multiband  video  sequences 
registered  by  the  TNO  TRI-band  Color  Low-light  OBServation  System  (TRICLOBS),  and  in  dynamic 
short-wave  infrared  (SWIR)  sequences  provided  by  AFRL/711HPW/RHCV.  In  addition,  we  applied  the 
method  for  the  early  detection  of  change  processes,  for  the  estimation  of  a  person's  heart  rate  from 
facial  videos,  and  to  enhance  the  visibility  of  landmines  in  polarized  SWIR  sequences.  In  the  course 
of  the  project  we  also  developed  a  simple  Power-Logarithm  (Pow-Log)  histogram  modification 
operator  to  enhance  image  contrast,  and  a  new  color-to-gray  conversion  algorithm  that  retains  both 
the  overall  appearance  and  the  discriminability  of  a  three  channel  color  image  when  represented  as 
a  single  band  (grayscale)  image. 


1  Objective 

The  objectives  of  the  project  "Enhancing  the  Detectability  of  Subtle  Changes  in  Multispectra  I 
Imagery  Through  Real-time  Change  Magnification"  (funded  by  Grant  FA9550-14-1-0069  )  were  to: 

•  Apply  newly  developed  Phase-Based  Video  Processing  (PBVP)  algorithms  to  enhance  the 
visibility  of  subtle  temporal  changes  (movement  or  temperature  fluctuations)  in  details  of 
interest  in  multispectral  imagery  while  simultaneously  reducing  dynamic  noise. 

•  Incorporate  these  algorithms  into  a  multiband  image  fusion  scheme. 

•  Demonstrate  the  practical  value  of  the  newly  developed  algorithms  by  applying  them  to  a 
set  of  example  videos. 


2  Results 

In  this  section  we  will  first  give  an  overview  of  the  results  of  applying  the  PBVP  method  to 
multispectral  imagery  (the  main  goal  of  this  project).  This  is  followed  by  an  account  of  some 
additional  supporting  algorithms  (an  adaptive  contrast  enhancement  algorithm  and  a  color-to- 
grayscale  transform)  that  were  developed  in  the  course  of  this  project. 

2.1  Motion  enhancement 

Recently  a  new  method  (called  Phase-Based  Video  Processing  or  PBVP)  was  introduced  to  enhance 
the  visibility  of  low-amplitude  temporal  (color  or  location)  changes  in  a  standard  video  sequences 
(Wadhwa  et  al.,  2013).  The  method  does  not  need  feature  tracking  or  optical  flow  computation  to 
amplify  time-varying  events,  but  merely  magnifies  temporal  phase  changes  through  hierarchical 
spatio-temporal  filtering.  A  spatially  oriented  complex-valued  multi-scale  (steerable  pyramid: 
Simoncelli  &  Freeman,  1995)  decomposition,  followed  by  temporal  filtering  is  applied  to  the  input 
frames.  The  complex  phase  variations  in  the  resulting  signal  are  then  either  amplified  to  reveal  any 
hidden  time-varying  information,  or  attenuated  to  remove  distracting  dynamic  noise.  The  method  is 
robust,  supports  large  magnification  factors,  operates  locally  and  is  not  sensitive  to  noise. 

In  this  study  we  adapted  the  PBVP  method  to  enhance  the  visibility  of  low-amplitude  temporal 
events  (i.e.,  of  subtle  changes)  in  multispectral  imagery. 
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The  PBVP  method  can  also  be  applied  to  reduce  dynamic  noise  by  attenuating  phase  variations  in 
the  complex  steerable  pyramid  transformed  input  signal. 


Finally,  it  is  in  principle,  possible  to  selectively  enhance  or  attenuate  temporal  events  at  different 
spatio-temporal  scales.  Thus,  movement  of  large  (e.g.,  vehicles)  and  small  (e.g.,  branches  or  leaves) 
details,  or  high  and  low  temporal  frequencies  (e.g.,  trembling  leaves  and  swaying  trees)  may  be 
selectively  enhanced  or  attenuated. 

PBVP  enhancement  can  be  applied  to  individual  bands  of  a  dynamic  multispectral  image  sequence. 
After  PBVP  enhancement ,  the  individual  bands  can  be  fused  into  a  single  dynamic  motion  sequence. 
Since  the  PBVP  enhancement  technique  employs  a  steerable  pyramid  filter  scheme,  which  is  highly 
suitable  to  fuse  multispectral  imagery  (Koren  et  al.,  1995;  Liu  et  al.,  2001),  it  can  in  principle  be 
incorporated  into  a  real-time  image  fusion  framework,  thus  integrating  the  enhancement  and  fusion 
into  a  single  scheme  (Toet  et  al.,  1989;  Toet,  1990). 


2.1.1  Motion  enhancement  of  TRI-band  Color  Low-light  OBServation  System  (TRICLOBS)  video 
sequences 

PBVP  enhancement  was  successfully  applied  to  enhance  the  visibility  of  small  movements  in  the 
Visual,  Near-Infrared  and  Thermal  (LWIR)  bands  of  multiband  TRICLOBS  video  sequences. 

One  of  the  examples  includes  a  TRICLOBS  movie  of  a  row  of  persons,  one  of  which  had  an  elevated 
heart  rate  and  was  panting  (he  had  been  running  just  before  the  video  was  recorded).  In  the  original 
movie  it  is  not  possible  to  visually  distinguish  the  person  with  the  elevated  heart  rate.  In  the  PBVP 
processed  movie  this  person  is  clearly  distinguishable  due  to  the  enhanced  movements  of  his  chest. 

Another  example  includes  a  TRICLOBS  movie  of  some  tall  trees.  While  these  appear  stationary  in 
the  original  footage,  they  are  clearly  swaying  in  the  wind  in  the  PBVP  enhanced  version  of  this 
movie. 

The  PBVP  motion  enhancement  appeared  to  work  well  on  all  the  individual  TRICLOBS  image  bands. 
After  enhancement,  the  individual  bands  can  be  combined  and  color-remapped  to  obtain  a  dynamic 
image  sequence  with  a  natural  color  appearance  (Hogervorst  et  al.,  2006;  Hogervorst  &  Toet,  2010). 


2.1.2  Motion  enhancement  ofSWIR  video  sequences 

PBVP  enhancement  was  also  successfully  applied  to  enhance  small  movements  in  dynamic  short¬ 
wave  infrared  (SWIR)  imagery  provided  by  Dr.  Alan  Pinkus  (AFRL/711HPW/RHCV).  The  method  was 
able  to  significantly  enhance  minor  movements  of  fingers,  chest,  eyes  and  arms  in  dynamic  SWIR 
sequences  of  a  person  who  was  appeared  static. 


2.1.3  Motion  enhancement  of  time-lapse  imagery 

We  also  tested  the  capability  of  the  PBVP  enhancement  for  the  early  detection  of  change  processes 
(e.g.,  diffusion  of  waste  material,  onset  of  fires,  onset  of  processes  like  disintegration  or 
decomposition).  The  idea  to  PBVP  enhance  dynamic  imagery  of  the  onset  of  such  processes  and 
play  the  enhanced  movies  in  a  looping  mode  to  enable  visual  detection  of  processes  that  would 
otherwise  go  unnoticed.  This  was  only  partly  successful.  The  problem  is  that  these  processes  are 
typically  restricted  to  a  small  part  of  the  image  in  their  early  stages.  As  a  result,  there  are  an 


Distribution  A:  Approved  for  public  release;  distribution  is  unlimited. 


insufficient  number  of  image  pixels  showing  a  change  and  the  method  is  ineffective.  Only  when 
these  processes  are  in  an  advanced  stage  does  the  method  enhance  their  amplitude,  but  by  then 
they  are  already  clearly  visible  in  video  sequences  that  are  not  enhanced. 


2.2  Heart  rate  estimation  from  facial  videos 

The  enhancement  of  temporal  color  variations  in  facial  videos  enables  the  extraction  of  a  person's 
heart  rate  (since  this  corresponds  to  the  frequency  of  the  facial  color  variations).  In  an  experiment 
we  simultaneously  measured  ECG  and  recorded  visual  imagery.  We  found  a  match  between  the 
heart  rate  derived  from  ECG  and  from  the  color  fluctuations  in  the  skin,  indicating  that  the  heart  rate 
can  be  derived  from  visual  imagery. 

2.2.1  Visualization  of  landmines  through  polarized  image  sequence  enhancement 

Previously,  a  3-5  pm  (mid-wave  infrared)  Radiance  high-speed  camera  in  combination  with  a  rotating 
polarization  filter  was  used  to  register  dynamic  imagery  of  landmines  that  were  left  undisturbed  for 
long  period  in  a  field  (Cremer,  2003).  As  a  result  the  landmines  were  largely  covered  with  tall  grass 
(i.e.,  they  were  obscured  from  direct  vision)  and  they  were  in  a  cluttered  environment  (vegetation, 
rocks,  etc.).  The  idea  is  that  the  representation  of  the  landmines  in  the  IR  sequences  will  show  a 
temporal  intensity  fluctuation  that  differs  from  the  clutter  elements  in  their  background. 

Deployment  of  PBVP  motion  enhancement  to  the  abovementioned  dynamic  video  sequences  did 
indeed  make  small  differences  in  the  temporal  variation  of  the  profiles  of  the  landmines  more 
salient,  thus  possibly  enabling  the  detection  of  landmines  that  would  otherwise  go  unnoticed. 

2.3  Adaptive  contrast  enhancement 

A  simple  Power-Logarithm  (Pow-Log)  histogram  modification  operator  was  developed  to  enhance 
infrared  (IR)  image  contrast  (Toet  &  Wu,  2015).  The  algorithm  combines  a  Logarithm  (Log)  operator 
that  smooth  the  input  image  histogram  while  retaining  the  relative  ordering  of  the  original  bins,  with 
a  Power  (Pow)  operator  that  restores  the  smoothed  histogram  to  an  approximation  of  the  original 
input  histogram.  Contrast  enhancement  is  achieved  by  using  the  cumulative  distribution  function  of 
the  resulting  histogram  in  a  standard  lookup  table  based  histogram  equalization  procedure.  The 
method  is  simple  and  independent  of  image  content  and,  unlike  most  existing  contrast 
enhancement  algorithms,  does  not  suffer  from  the  occurrence  of  artifacts,  over-enhancement,  and 
unnatural  effects  in  the  processed  images.  The  method  can  be  applied  both  in  a  Direct  Pow-Log 
(DPL)  and  in  an  Iterative  Pow-Log  (IPL)  mode. 

Objective  (computational)  and  subjective  (visual)  evaluation  studies  using  a  wide  range  of  different 
IR  input  images  showed  that  DPL  and  IPL  both  significantly  enhance  image  contrast  while  retaining 
the  overall  image  structure  and  preserving  the  perceptibility  of  small  details  and  targets  (Toet  &  Wu, 
2015).  DPL  retains  image  structure  slightly  better  than  IPL,  while  IPL  enhances  contrast  slightly  more 
than  DPL.  Experimental  results  proved  that  the  proposed  method  is  a  robust  and  efficient  technique 
for  effective  IR  image  contrast  enhancement. 

Using  a  wide  range  of  different  color  input  images  it  was  shown  that  the  power-logarithm  contrast 
enhancement  method  can  also  be  applied  to  regular  color  images  (e.g.,  photographs),  and  achieves 
appreciable  contrast  enhancement  on  under  diverse  lighting  conditions  (Toet  &  Wu,  2014;  Wu  & 
Toet,  2014a).  Moreover,  performance  comparisons  with  several  state-of-the-art  contrast 
enhancement  algorithms  showed  that  the  method  performs  comparably  or  even  better  than 
computationally  more  complex  and  time  consuming  methods. 
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2.4  Color-to-grayscale  conversion 

We  developed  a  new  color-to-gray  conversion  algorithm  that  retains  both  the  overall  appearance 
and  the  discriminability  of  the  three  channel  (in  the  rest  of  this  report  denoted  as  RGB)  input  image 
(Wu  &  Toet,  2014b).  Actually,  the  input  image  may  consist  of  more  than  three  channels,  which  need 
not  correspond  to  realistic  colors  but  may  contain  input  from  imaging  sensors  operating  in  different 
spectral  bands  (even  outside  the  visual  spectrum).  The  algorithm  uses  a  weighted  pyramid  image 
fusion  scheme  to  blend  the  R,  G,  and  B  color  channels  of  the  input  image  into  a  single  grayscale 
image.  First,  a  scalar-valued  weight  map  is  computed  for  each  of  the  R,  G,  and  B  color  channels. 

This  weight  map  is  composed  of  several  simple  visual  quality  metrics  and  serves  to  preserve  only  the 
most  salient  (informative)  visual  details  from  each  of  the  input  color  channels  in  the  final  grayscale 
image.  Next,  a  multiresolution  weight  map  is  constructed  for  each  channel  by  applying  a  Gaussian 
pyramid  transform  to  its  corresponding  weight  map.  The  three  multiresolution  weight  maps  are 
then  normalized  such  that  they  sum  to  one  for  each  pixel.  Then,  the  R,  G,  and  B  color  channels  of 
the  input  image  are  decomposed  into  Laplacian  pyramids,  which  basically  contain  band-pass  filtered 
versions  at  different  spatial  scales.  Finally,  after  multiplication  of  these  Laplacian  pyramids  by  their 
corresponding  Gaussian  pyramid  weight  maps  and  summation,  the  final  grayscale  converted  image 
is  obtained  by  reconstructing  the  resulting  pyramid.  The  method  is  straightforward  and  automatic 
(requires  no  user  interaction),  has  limited  complexity,  and  performs  at  least  as  well  as  the  best 
performing  state-of-the-art  color-to-grayscale  conversion  algorithms.  The  main  contribution  of  this 
method  is  that  it  formulates  the  color-to-grayscale  conversion  problem  as  a  visual  saliency  weighted 
multi-scale  channel  fusion  scheme,  to  achieve  a  grayscale  representation  that  optimally  represents 
the  information  contained  in  each  of  the  channels  of  an  RGB  color  image. 


3  Deliverables 

3.1  Software  and  imagery 

The  algorithms  that  were  used  and  developed  in  the  course  of  this  project  were  programmed  in 
Matlab  2012a.  All  Matlab  code,  images  and  video  sequences  used  in  this  study  will  be  provided  to 
Dr.  Alan  Pinkus  (AFRL/711HPW/RHCV). 

3.2  Publications 

The  following  publications  resulted  directly  from  this  study  and  all  contain  the  appropriate 
acknowledgments  to  Grant  FA9550-14-1-0069  : 

•  Toet,  A.  &  Wu,  T.  (2014).  Efficient  contrast  enhancement  through  log-power  histogram 
modification.  Journal  of  Electronic  Imaging,  23(6),  063017.  DOI  10.1117/1.JEI.23.6.063017. 

•  Toet,  A.  &  Wu,  T.  (2015).  Infrared  contrast  enhancement  through  log-power  histogram 
modification.  Journal  of  Pattern  Recognition  Research,  10(1),  1-23.  DOI  10.13176/11.617. 

•  Wu,  T.  &  Toet,  A.  (2014a).  Efficient  contrast  enhancement  through  log-power  histogram 
modification.  In  D.A.  Huckridge  &  E.  Reinhard  (Eds.),  Electro-Optical  and  Infrared  Systems: 
Technology  and  Applications,  SPIE-9249-26  (pp.  1-14).  Bellingham,  WA,  USA:  SPIE. 
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•  Wu,  T.  &  Toet,  A.  (2014b).  Color-to-grayscale  conversion  through  weighted  multiresolution 
channel  fusion.  Journal  of  Electronic  Imaging,  23(4,.  article  no.  043004),  1-6.  DOI 
10.1117/l.JEI. 23.4.043004. 


4  Concluding  Remarks 

All  results  and  the  deliverables  will  be  presented  to  Dr.  Alan  Pinkus  (AFRL/711HPW/RHCV)  and  his 
colleagues  during  a  Windows  on  Science  visit  by  Dr.  Alexander  Toet  and  Dr.  Maarten  Hogervorst  to 
Wright-Patterson  AFB,  OH  ,  from  27-29  May  2015. 
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